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ABSTRACT 


Prior research has shown that, under certain conditions, 
Human-Agent (H-A) alignment exists to a stronger degree 
than that found in Human-Human (H-H) communication. 
In an H-H Second Language (L2) setting, evidence of align- 
ment has been linked to learning and teaching strategy. We 
present a novel analysis of H-A and H-H L2 learner dialogues 
using automated metrics of alignment. Our contributions 
are twofold: firstly we replicated the reported H-A align- 
ment within an educational context, finding L2 students 
align to an automated tutor. Secondly, we performed an 
exploratory comparison of the alignment present in compa- 
rable H-A and H-H L2 learner corpora using Bayesian Gaus- 
sian Mixture Models (GMMs), finding preliminary evidence 
that students in H-A L2 dialogues showed greater variability 
in engagement. 
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1. INTRODUCTION 


This work reports on evidence of alignment within student 
dialogue to that of an automatic tutor even when both par- 
ties are restricted in their capacity to align: the student as 
an L2 learner may lack the linguistic proficiency to show 
alignment [5], and the agent aligns only minimally by de- 
sign. Alignment consists of interlocutor interaction adap- 
tation, resulting in convergence, or in their sharing of the 
same concept space [13, 8]. Alignment of student to tutor 
in dialogue has been used as a predictor of both student 
learning and engagement [20]. A key aspect of dialogue is 
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the speakers’ ability to align: to either show engaged, will- 
ing behaviour, or display little discernible adaption to their 
interlocutor. Interestingly, humans have been shown to ex- 
hibit greater alignment to agents than to other humans (4, 
6]. In an automated L2 tutoring setting, where students 
have been shown to imitate tutors as part of their learning 
process [10] it is of great interest to determine whether the 
user/learner is actively engaged, simply gaming the system, 
or disengaged, either because of lack of ability or motiva- 
tion [1]. Modelling alignment of student to tutor as evidence 
of engagement could serve as a useful tool in the design of tu- 
tor intervention or student assessment since there has been 
limited research into identifying signs of engagement or gam- 
ing in the automated L2 tutoring setting. 


Given this relevance of alignment in modelling engagement 
during tutor-student L2 dialogues [20], one key question is 
whether L2 students demonstrate alignment behavior in con- 
versation with an automated dialogue agent, even when they 
know the agent is not human. Prior work has established 
that L2 students display alignment when conversing with 
a human tutor, in Human-Human (H-H) interactions [17]; 
however, this work has also demonstrated relatively symmet- 
ric alignment, as human tutors verbally aligned with their 
students in turn — this raises the possibility that L2 learn- 
ers may fail to display alignment if the dialogue is predom- 
inantly asymmetric , when interacting with a agent whose 
capacity to align is also limited. Studies of Human-Agent 
(H-A) dialogues in other domains demonstrate that fluent 
speakers verbally align with agents [4, 6], but given the 
unique constraints affecting alignment in L2 dialogue [5], 
we cannot assume that L2 students will behave similarly. If 
they do, a second key question arises: do L2 students display 
similar alignment behavior in H-H and H-A dialogues? Even 
if students align in both contexts, exploratory analysis may 
reveal critical differences which could inform educational re- 
searchers and practitioners working with dialogue agents. 
Hence, our work addresses the following research questions: 
RQ1 Do L2 students show alignment to an automated di- 
alogue agent (i.e. H-A alignment)? and RQ2 What is the 
nature of the alignment found in the H-A corpus and how 
does it differ from that of H-H dialogues? 
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We present a study of student verbal alignment within a new 
dialogue corpus consisting of transcripts from a language 
teaching app where students are interacting with a dialogue 
agent. We contrast this H-A corpus with a comparable H- 
H L2 learner corpus of tutoring dialogue transcripts. We 
found that students in H-A interactions align to the agent 
more so than they would by chance, albeit to a lesser de- 
gree than students in H-H dialogues. Our results found that 
within H-H dialogues, students exhibited greater alignment 
than tutors. Finally, we compared the distribution of stu- 
dent to tutor alignment within both corpora, revealing more 
variance in alignment within the H-A dialogues. We hypoth- 
esise this was due to either student engagement effects, or 
different types of student alignment strategy within the H-A 
dialogues than the more uniform alignment present in the 
H-H corpus. 


2. BACKGROUND 


To achieve effective communication within dialogue, speak- 
ers typically align, adapting their interaction to their inter- 
locutor. The Interactive Alignment Model (IAM) [13], de- 
scribes this process as that of speakers agreeing on a shared 
conceptual space. In educational settings, by contrast, align- 
ment has been found to predict both student learning and 
engagement [20].Automatic alignment between interlocutors 
occurs over different linguistic levels, including that of the 
lexical, syntactic and semantic [13]. Lexical alignment con- 
sists of speakers beginning to use the same words [21, 17] 
or phrases [6] as each other. Syntactic alignment consists of 
the use of the same parts of speech patterns, such as similar 
noun-phrase constructions, or similar adjuncts [14] as the 
conversation progresses. Finally, semantic alignment can 
range from adaptation to individual differences in person- 
ality [11] to convergence at a higher level of representation 
such as Dialogue Acts [16]Recent research has established 
a number of metrics for linguistic alignment which can be 
computed automatically, enabling large-scale corpus analy- 
sis based on sequential pattern mining [6]. These methods 
quantify alignment in terms of the expressions, or contigu- 
ous sequences of tokens appearing in the utterances of both 
interlocutors. While these methods have been applied to the 
analysis of H-A interaction [6] and H-H student-tutor inter- 
action [17], the work presented in this paper is the first to 
apply this computational methodology to compare H-A and 
H-H dialogue in an educational L2 setting. 


Within an L2 practice setting, we predict alignment to have 
slightly different properties compared to a fluent conversa- 
tional setting where speakers tend to have a symmetric con- 
tribution and equal status within the dialogue [18], and are 
equally capable of participating [5]. L2 learners have been 
found to perform at a higher level when speaking in dia- 
logue with a peer than in a monologue context [15] This 
suggests students draw from the example language of their 
interlocutor leading us to expect evidence of alignment. L2 
students have also been shown to learn vocabulary through 
taking part in dialogue [9], suggesting this process of align- 
ment and repetition of their interlocutor’s speech produces 
learning gains. In the case of the tutor, their need to adhere 
to the ZPD suggests that their alignment patterns will also 
differ from that of straightforward dialogue. These different 
factors influence the speakers’ convergence to a shared men- 
tal state [5]. Vygotsky’s theory of ZPD [19] states students 


Table 1: Tutorbot dialogue example. Italics indicate 
Expression Repetition 


1. bot: What is your favorite day of the week ? 
2; user: | My favorite day of the week is Friday ... 
3. bot: Do you play sports ? 

4, user: | yes 

5. bot: What sport do you play ? 

6. user: | I play volleyball and I go running 

ts bot: When do you do that ? 

8. user: | On Monday , Wednesday and Friday 

9. bot: What time does it start ? 

10. user: | At 4 o’clock in the afternoon 


will learn best when addressed at the correct level, therefore 
we also expect to see alignment, in the case of tutors in H-H 
dialogues, to student ability. 


3. CORPORA 


We are interested in the comparison between student align- 
ment in H-H and H-A dialogues. The H-A corpus analyzed 
in this study comprises dialogues drawn from a large-scale 
commercial platform for L2 learners’. In this application, 
novice learners of English who had completed lessons on rele- 
vant topics were offered the possibility to review the material 
via simple conversations with the automated dialogue agent 
Tutorbot. Given the focus on relevant learning material, 
the agent engaged learners in a system-initiative dialogue 
with extensive guidance, rather than user-initiative [2]; as 
a result, Tutorbot steered the learner conversations very de- 
liberately, and alignment from the tutor agent to the student 
was highly limited by design. A sample dialogue from the 
corpus can be seen in Table 1. The H-H corpus used is 
the Barcelona English Language Corpus (BELC) [12] which 
consists of tutor guided conversations with L2 learners of En- 
glish at varying stages of fluency from absolute beginner to 
approaching intermediate. The tutor’s goal was to elicit as 
much conversation from the learner as possible while setting 
them at ease in as natural and conversational a manner as 
they could. Key differences are shown in Table 2. However, 
it should also be noted that the Tutorbot corpus only con- 
sists of single utterance turns, whereas BELC has multiple. 
The topics are also more diverse in BELC, as the Tutor- 
bot explicitly guided learners to review practiced material 
rather than engage in open-ended discussion. Nevertheless, 
certain main topics (how are you, where are you from, tell 
me about your family, hobbies, what time do you do that) 
and the beginner/lower-intermediate range of learner abil- 
ity are common to both, facilitating automated alignment 
comparison. 


4. METHODS 
4.1 Alignment 


In order to analyse the verbal alignment present in both 
corpora, which allows us to answer both RQ/ and RQ2, we 
use the expressions-based measures introduced by [6]. This 
approach identifies sequences of tokens (Expressions) which 
are used by both dialogue participants (thus established as 
expressions). These expressions allow us to see the fixed ex- 
pressions established between speakers, called the routiniza- 


'This data was kindly shared with us by Babbel, 
https: //www.babbel.com/ 
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Table 2: H-A and H-H Corpora Differences 


Tutorbot BELC 
number of dialogues 3689 118 
average Num. utterances 20.41 130.69 
average Num. tokens 128.99 634.28 
average tokens/utterance 6.32 4.85 
communication medium typed spoken 
speakers H-A H-H 
student L1 German | Spanish 
vocabulary overlap 0.085 0.251 


tion process in the interactive alignment theory [13], and 
thus an indication of speaker alignment. We re-define the 
following in order to discuss our results in the following sec- 
tions: 


Expression Lexicon EL is the set of expressions used by 
both speakers for a given dialogue. 


Expression Variety (EV) is the size of the EL normalised 
by the total number of tokens in the dialogue. This ra- 
tio indicates the variety of the expression lexicon rela- 
tively to the length of the dialogue: the higher the EV, 
the more incidence of established expressions between 
participants. The EV indicates the routinization be- 


tween speakers. 
EV = length(EL) 


numberoftokens 


Expression Repetition — speaker (ERs) is the ratio of 
Expressions to dialogue produced. This is measured in 
tokens. This value indicates the Expression repetition 
present in the dialogue, i.e. the higher the ER, the 
more the speakers dedicate tokens to the repetition of 
established expressions. This is indicative of speaker 
alignment. 


Initiated Expression (IEs) are the established expressions 
initiated by 5 


Vocabulary Overlap (VO) is the ratio of shared tokens 
between interlocutors $; and Se. The higher the VO, 


the more vocabulary is shared between speakers. 
VO- (Tokensg, NTokensg, ) 


(Tokensg, UTokenssg, ) 


4.2 Baseline 


In order to test that the alignment reported was not sim- 
ply due to corpus-specific vocabulary effects (which would 
be influenced by the vocabulary overlap defined in the pre- 
vious section), a ‘scrambled baseline’ was created for each 
corpus. This was achieved by creating a ‘bag of words’ of 
the tokens produced by each speaker for a specific dialogue, 
then substituting each token from each speakers utterances 
with one from the shuffled bag of words. This method re- 
tains the turn-taking of the speakers, and the distribution 
of utterance lengths from the original dialogue, but removes 
any word ordering present. In the results section for each 
alignment measure, we report on whether the effects were 
significantly different from this baseline. This baseline al- 
lows us to compare the effects of alignment across corpora, 
answering RQ1. 


4.3 Alignment Distribution Clustering 

In order to answer RQ2 investigating student alignment dif- 
ferences within and between the H-H and H-A corpora, we 
fitted a Gaussian mixture model (GMM)|[7] to the student 
ERs data for both the H-H and H-A students. GMMs al- 
lowed us to detect and characterize distinct sub-populations 
within a larger group, provided those sub-populations were 
marked by differences in a parameter of interest, e.g., mea- 
sured ERs. To find the number of components which best 
fitted the data, we used a Bayesian Gaussian mixture model 
with a Wishart prior of [[0.1]] on the precisions and a scale-1 
exponential prior on the number of clusters, and selected the 
most probable number of clusters given the data (i.e. the 
posterior mode), assuming that up to seven clusters might 
be present. We used a Bayesian approach in order to avoid 
the degeneracies that are common when using maximum- 
likelihood estimation and information criteria (e.g., AIC or 
BIC) to estimate cluster counts and parameters [3]. To 
implement this, we used the toolkit scikit-learn?, package 
BayesianGaussianMizture; the priors on component means 
were scikit-learn 0.20 defaults. 


5. RESULTS AND ANALYSIS 


The following subsections all contribute to answering RQJ, 
through the comparison of H-H to H-A student alignment 
and corpus statistics. Section 5.5 specifically explores the 
variation in alignment styles across corpora, allowing us to 
answer RQ2. 


5.1 Expression Lexicon 

The Expression lexicon is the set of expressions which are 
shared between speakers. On inspection, the most common 
multi-word expressions being aligned to in the Tutorbot cor- 
pus fell into two main categories: 1) the student using the 
direct re-form of the question in the creation of their answer: 
“bot|4: What is your favorite day of the week ? user|éd: 
My [favorite day of the week] [is] Friday”. 2) The student 
reflecting the question back to the tutor-bot. “bot|4: Where 
do you live? user|5: I live in <LOCATION>, where [do 
you live/?”. The rephrasing in BELC is different: it is more 
likely that the tutor will re-phrase the student’s single or 
multi word answer as a form of confirmatory feedback. e.g. 
“Tutor: you like going out with your friends, good” when this 
is really more repetition/confirmation. The student align- 
ment also consisted of their reflection of tutor questions back 
to them, and in their repetition of tutor scaffolding moves 
(something not present in the Tutorbot corpus due to the 
agent dialogue design) Table 3 contains details of the vocab- 
ulary overlap, speaker specific token ratios and the expres- 
sion lexicon size differences between corpora. 


Table 3: Corpora Differences- values represent the 
average per dialouge 


Tutorbot | BELC 
Expression Lexicon Size (ELS) 3.04 48.55 
S1/tokens (%) 0.81 0.68 
S2/tokens (%) 0.19 0.30 
Voc. Overlap 0.085 0.251 
Voc. Overlap $1 0.105 0.312 
Voc. Overlap $2 0.258 0.613 


“https: //scikit-learn.org/stable/ 
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5.2 Vocabulary Overlap 

The vocabulary overlap (VO) between speakers gives us an 
idea about how likely ‘’alignment’ according to our metric 
will occur by chance. The results in Table 3 therefore can 
inform our interpretation of the levels of ERs reported in 
section 5.4. Student VO in BELC (HH) is much higher than 
from the students in Tutorbot (HA) (0.613 vs. 0.258) This 
could be due to the fact that Tutorbot learners were at a 
lower level of proficiency, so they did not use such extensive 
vocabulary; alternatively, it could be due to the method of 
data collection: Tutorbot allows learners a one turn response 
(a single utterance), limiting their production. 


Corpora comparison for speaker Vocabulary overlap} 


104 


0.84 


0.64 


Ratio 


0.47 


0.04 


T r T T 
HAISL HAIS2 HH/SL HH/iS2 
Speaker 


Figure 1: H-H/A corpora Vocabulary Overlap. 
Speaker difference was significant for H-A (p < 
0.0001 (statistic = 6.42, pvalue = num1.4e — 10) and H- 
H (p < 0.001) (statistic = —2.11, pvalue = 0.00036) S1 
= Tutor/Agent, S2 = Student 


5.3. Expression Variation 

We compare the H-H and H-A corpora of real interactions 
to each other, and to the baseline H-Hp and H-Ar corpora 
to control for vocabulary effects. Firstly, EV was signif- 
icantly higher for the H-H corpus (mean = 0.075, std = 
0.025) than that in the H-A corpus (mean = 0.032, std = 
0.046). Statistical difference was checked by performing a 
t-test (statistic = —10.05, p — value = 1.888 x 1077), indi- 
cating H-H interactions result in a richer expression lexicon 
than H-A interactions. The EV values were much lower than 
those reported for negotiation dialogues [6], which may be 
due to dialogue type: routinisation may form a much greater 
part of negotiation than it does L2 tutoring. Another reason 
for the low EV in the H-A corpus is that the student cannot 
establish expressions other than by chance since the Tutor- 
bot corpus is system-initiated and is not designed to align 
to the student’s responses. Neither the EV of the H-H nor 
the H-A corpus was statistically greater than the H-Hp and 
H-Ap baselines, which can be in part attributed to the high 
proportion of single-token expressions in both corpora, lead- 
ing to greater likelihood of their existence in the scrambled 
baseline. 


5.4 Expression Repetition 
Expression repetition (ERs) is the main indication of speaker 
alignment measured. Figure 2 shows the different degrees 


Corpora comparison for speaker Expression Repetition {ER} 
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Figure 2: H-H/A corpora ERs. Speaker differ- 
ence is significant for H-A (p < 0.0001) (statistic = 
—44.91, pvalue = 0.0) and H-H (p < 0.0001) (statistic = 
—12.71, pualue = 1.77 x 10~*) S1 = Tutor/Agent, S2 = 
Student 


of ERs for both the H-A and H-H corpora. The differ- 
ence between the ERs of each speaker was significant for 
both corpora: H-A (statistic = —44.91,p — value = 0.0) 
and H-H (statistic = —12.71,p — value = 1.770 x 107*8). 
It is interesting to note the asymmetry between speakers 
for both dialogues. The tutor in the H-H dialogues had 
a significantly lower proportion of ER than the student, 
suggesting ER has less to do do with teacher strategy as 
with learner strategy. We compared each ERs with its 
ERr for both corpora: for the H-A corpus, student ERs2 
(mean = 0.192, std = 0.235) was significantly higher than 
that of ERrs2e (mean = 0.134, std = 0.206) (statistic = 

11.20, p — value = 6.593 x 10~7°). Meanwhile, tutor ERsi 
(mean = 0.016, std = 0.032) was significantly lower than 
that of their scrambled baseline ERr_si (mean = 0.024, std = 
0.037) (statistic = 9.865,p — value = 8.2012 x 10773) in- 
dicating the absence of alignment expected from an agent 
not designed to do so. For the H-H corpus, student ERs2 
was not significantly different from their baseline ERr_s2 
(statistic = 0.932, p — value = 0.352), nor was tutor ERs1 
(statistic = 2.506,p — value = 0.013). This can be ex- 
plained in part by the fact that VO for the H-H corpus 
(mean = 0.251, std = 0.061) was significantly larger than in 
the H-A corpus (mean = 0.085, std = 0.146) (statistic = 

12.32, p — value = 3.089 x 107°"), 


5.5 Student ER Distribution 


In answer to RQ2, we compare the distributions of per- 
dialogue ERs values between H-A and H-H corpora. Fig- 
ure 3 shows histograms of ER frequency for each corpus, 
which suggest there were multiple types of student align- 
ment in the H-A corpus (a), in contrast to a single cluster of 
ER values for the H-H corpus (b). To quantify these differ- 
ences in student alignment — and go beyond a comparison of 
averages which neglects the possibility of differences across 
individuals and dialogues — we fit a Bayesian Gaussian Mix- 
ture Models [7] (described in Section 4.3) to student ERs 
values. The results of our model indicate that the most 
probable number of clusters, given the data (i.e., the pos- 
terior mode), was 5 for the H-A corpus (Figure 3a) and 1 
for the H-H corpus (Figure 3b). This analysis also reveals a 
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Figure 3: 


Frequency of Expression Repetition values (High ER indicates greater alignment). 
Mixture Models (GMM) which best fitted to the data shown by the red line. 


mean: 0.33 
weight: 1 


Gaussian 


Means: centroids of the 


component clusters. Weight: the proportion of dialogues in a cluster. 


Table 4: Qualitative Analysis of H-A dialogues at 
the ‘centroids’ of the component clusters 


ER Description and example 

0-0.01 no-response or request for help: students ei- 
ther do not engage with the agent, or demonstrate 
inability to engage 

0.1-0.15 minimal response: students respond curtly, ap- 
pear less engaged 
bot: What is your favorite day of the week ? 
user: That ’s Sunday . 

0.25-0.4 high engagement: dialogues either longer with 
align and rephrase within longer utterances, with- 
out excess repetition, or shorter dialogues consist 
of more repetition and rephrasing, and the limited 
vocabulary contributes to alignment 
bot: Do [you] _have a_ boyfriend or _a_ girlfriend 
? Or _a husband_ or _a_ wife ? 
user: I [have [a] husband] . 

0.5-0.55 minimal response: low rate of student produc- 
tion, typical response one high-frequency word, 
low engagement despite high alignment 
hi’, bye. , 

0.85-0.9 high repetition: ll student responses are 
rephrases, dialogues very short 
bot: _Hello_ , _nice to see you. ! 
user: [Hello] [nice to see you] too 


cluster in the H-A corpus which has a qualitatively compa- 
rable mean value to the one in H-H (0.310-H-A, 0.330-H-H). 
Table 4,shows this cluster contains the longest dialogues in 
Tutorbot, which are qualitatively the most similar to those 
in BELC. 


We hypothesise the other clusters are either, in the case 
of low level ER, signs of student lack of engagement (align- 
ment being symptomatic of engagement within dialogue) or, 
in the case of higher ER, signs that the students are in some 
way conversing in a manner impossible to find in H-H dia- 
logues. We hypothesise either this is due to the communi- 
cation medium: students can copy, paste and edit the agent 
utterance to create their response or due to students’ desire 
to learn through continual repetition of the agent’s phrases. 


Table 4 shows examples and descriptions of the H-A corpus 
data, corresponding to the component means in Figure 3. 
Since the H-H corpus was gathered as part of an experi- 
ment, we know that there would not be ‘outlier’ behaviour 
present, but the upper and lower ranges show some differ- 
ences in interaction style of the learner. 


6. DISCUSSION 


In relation to RQ1, whether there is evidence of student 
- agent alignment in L2 dialogues, we find significant H-A 
alignment. The magnitude of this effect was weaker than 
that found in H-H dialogues, and we hypothesise that adap- 
tive student support in the form of tutor alignment is es- 
sential for students to align to the degree they do in an L2 
H-H setting. We found no significant alignment of agent to 
student, however an agent designed to interact with more 
explicit alignment may more resemble the alignment found 
in the H-H corpus. We found asymmetrical alignment within 
the H-H corpus, which was in keeping with results reported 
on lexical priming for the same corpus which found the 
strongest priming effects are those from student to tutor [17]. 
In relation to RQ2, concerning the exploratory analysis of 
alignment differences across corpora, a particularly salient 
finding are the differences in alignment across dialogues, 
suggesting different patterns of student engagement could 
be detected via their alignment levels. Table 4 shows that 
there was a clear ‘normal range’ for interaction, and the 
outliers showed different signs of student non-engagement. 
Our key finding is that there was greater variability in H- 
A compared to H-H alignment (best fit of 5 clusters com- 
pared to a single cluster), although role of factors such as 
dialogue and utterance length in these findings should be 
investigated in future work. We hypothesise that building 
a more alignment-focused tutoring agent could increase stu- 
dent engagement and yield results consistent to those within 
BELC. This could lead to better online L2 tutoring systems 
which promote student engagement and therefore improve 
participation and learning. It may be that the nature of 
an online learning platform will always result in some stu- 
dents who do not fully engage, and need different interven- 
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tion strategies. Using an alignment metric in the manner 
of our study could allow for the identification of these stu- 
dents, measurement of their engagement, and prediction of 
personalised interventions. 


7. CONCLUSIONS AND FUTURE WORK 


This paper presents a comparative analysis on student to tu- 
tor alignment in both an H-A and an H-H dialogue setting. 
We found students aligned to the agent, although this align- 
ment was not stronger than that present in H-H dialogues 
which is the case for both negotiation [6] and task-based di- 
alogues [4]. We hypothesise we can better explore this in a 
setting where the agent is specifically designed to align to 
the student.A limitation of our study is that both corpora 
were collected independently and therefore differ in more 
aspects than the one we wish to explore. In future work 
it would be desirable to collect data in a controlled setting 
which is more similar to the Tutorbot corpus to facilitate 
a more in-depth comparison. Another avenue for future re- 
search is the design of adaptive ‘alignment’ moves for the 
automated tutor to make. The design could draw on how 
the ZPD influences alignment and what the common ERs 
are in the H-H corpus, such as confirmatory rephrasing (e.g. 
“Student: I speak Germanish”, “Tutor: you speak German? 
Great!”) or repetition (e.g. “student: I am 20 years old”, 
“tutor: 20 years old? good!”). This research has a number 
of implications for the educational community, particularly 
regarding the use of alignment as an indicator of engage- 
ment. Furthermore, our method of clustering student ERs 
to identify ‘normal’ engagement behaviour for a given do- 
main may inform the detection of outliers and has potential 
for automating dialogue planning and intervention policies. 
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